RDD - Create RDD
Creating an RDD is the starting point of any Apache Spark application. Spark provides several methods to construct RDDs, depending on where your data originates.
This guide catalogs the four primary methods for creating RDDs in PySpark, complete with detailed code examples.
The Four Methods of Creating RDDs
graph TD
A["How to Create an RDD?"] --> B["1. From Memory Collection (sc.parallelize)"]
A --> C["2. From External Files (sc.textFile / sc.wholeTextFiles)"]
A --> D["3. Creating an Empty RDD (sc.emptyRDD)"]
A --> E["4. Deriving from Existing RDDs (Transformations)"]
style A fill:#fff3e0,stroke:#e65100,stroke-width:2px;
Setting Up Spark Session (For Code Examples)
Ensure you have your environment initialized before running the examples:
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName("Day01 Ways to Create RDD") \
.master("local[*]") \
.getOrCreate()
sc = spark.sparkContext
1. From a Local Memory Collection (sc.parallelize)
This method converts a pre-existing list or iterable in the Driver program's memory into a distributed dataset.
- When to use: Prototyping, unit testing, or defining small lookup reference tables.
- Key parameter:
numSlices(specifies partition count).
# 1. Local list of string values
programming_languages = ["Python", "Scala", "Java", "R", "SQL"]
# 2. Convert to distributed RDD with 3 partitions
languages_rdd = sc.parallelize(programming_languages, numSlices=3)
# 3. View the properties
print("Created RDD Type:", type(languages_rdd))
print("Number of Partitions:", languages_rdd.getNumPartitions())
print("Elements:", languages_rdd.collect())
# Output:
# Created RDD Type: <class 'pyspark.rdd.RDD'>
# Number of Partitions: 3
# Elements: ['Python', 'Scala', 'Java', 'R', 'SQL']
2. From External Storage Systems (sc.textFile)
Loads data files located on a local disk, an HDFS path, an S3 bucket, or Azure Blob storage.
- When to use: Reading production log files, database dumps, CSV, or raw text streams.
- Key parameter:
minPartitions(suggests minimum partitions, defaults to 1 per 128MB HDFS block).
# 1. Read a text file from local path
# Spark will read each line as a separate element in the RDD
file_rdd = sc.textFile("sample_logs.txt", minPartitions=2)
# 2. Output stats
print("RDD from File - Partitions:", file_rdd.getNumPartitions())
print("First Line of File:", file_rdd.first())
3. Creating an Empty RDD
Sometimes you need to initialize a completely blank RDD to act as a placeholder. This is highly useful in loops, recursive algorithms, or when you conditionally union datasets based on complex program states.
Spark offers two ways to create a blank RDD:
Method A: sc.emptyRDD()
Creates a completely empty RDD with no partitions and no elements.
# 1. Create a completely empty RDD
empty_placeholder_rdd = sc.emptyRDD()
# 2. Inspect properties
print("Empty RDD Type:", type(empty_placeholder_rdd))
print("Number of Partitions:", empty_placeholder_rdd.getNumPartitions()) # 0
print("Is RDD Empty?", empty_placeholder_rdd.isEmpty()) # True
# Output:
# Empty RDD Type: <class 'pyspark.rdd.EmptyRDD'>
# Number of Partitions: 0
# Is RDD Empty? True
Method B: Parallelizing an Empty List (sc.parallelize([]))
Creates an RDD with no elements, but retains partitions (determined by numSlices or defaults). This is useful if you want to partition-align an empty placeholder before joining or unioning.
# 1. Create empty RDD with 4 active empty partitions
empty_with_partitions_rdd = sc.parallelize([], numSlices=4)
# 2. Inspect properties
print("Empty partitioned RDD Type:", type(empty_with_partitions_rdd))
print("Number of Partitions:", empty_with_partitions_rdd.getNumPartitions()) # 4
print("Is RDD Empty?", empty_with_partitions_rdd.isEmpty()) # True
# Output:
# Empty partitioned RDD Type: <class 'pyspark.rdd.RDD'>
# Number of Partitions: 4
# Is RDD Empty? True
4. Deriving an RDD from an Existing RDD
Due to RDD immutability, when you apply a transformation (like map, filter, or reduceByKey) to an existing RDD, Spark does not modify the source; instead, it generates and returns a brand new RDD with a child reference to the parent.
- When to use: Building modular processing pipelines.
# 1. Source RDD
source_rdd = sc.parallelize([1, 2, 3, 4, 5])
# 2. Derived RDD A: Filter for even numbers (returns a new RDD)
evens_rdd = source_rdd.filter(lambda x: x % 2 == 0)
# 3. Derived RDD B: Square the filtered numbers (returns another new RDD)
squared_evens_rdd = evens_rdd.map(lambda x: x * x)
# 4. Fetch the final results
print("Source Data:", source_rdd.collect()) # [1, 2, 3, 4, 5]
print("Squared Evens Data:", squared_evens_rdd.collect()) # [4, 16]